Image processing apparatus and method

ABSTRACT

The present disclosure relates to image processing apparatus and method which enable a sense of quality as if there were a real object to be more favorably reproduced. A characteristic control unit determines processing contents for controlling the sense of quality, for example, changing an image quality, adjusting a reflection characteristic, or changing a shape, on the basis of a physical characteristic parameter from a characteristic information integration unit and device information from a device information analyzer. The characteristic control unit supplies information on the determined processing contents to an image composition unit and an additional-information generation unit and causes the image composition unit and the additional-information generation unit to perform sense-of-quality control. That is, in the image processing apparatus, more realistic visual expression becomes possible with an image output of one device as well as an output other than an image from another device. The present disclosure can be applied to the image processing apparatus, for example.

TECHNICAL FIELD

The present disclosure relates to image processing apparatus and methodand more particularly to image processing apparatus and method whichenable a sense of quality as if there were a real object to be morefavorably reproduced.

BACKGROUND ART

For the purpose of improving the realisticness of videos, technologiesof adjusting the contrast, the fineness, and the like have beendeveloped.

Note that Patent Literature 1 has proposed a technology to realize acolor correction and desired color reproduction of an object in an imageand improve the image quality.

CITATION LIST Patent Literature

-   -   Patent Literature 1: WO2010/087162

DISCLOSURE OF INVENTION Technical Problem

However, it has been difficult to further improve the performance onlyby improving the basic image quality such as the contrast and thefineness. Thus, it is desirable to reproduce a sense of quality as ifthere were a real object, for example, visually reproduce a situation asif an object were actually seen.

The present disclosure has been made in view of the above-mentionedcircumstances to be capable of more favorably reproducing a sense ofquality as if there were a real object.

Solution to Problem

An image processing apparatus according to an aspect of the presentdisclosure includes: a physical-characteristic parameter acquisitionunit that acquires a physical characteristic parameter regarding anobject of an image; a sense-of-quality control unit that controls asense of quality of the object in the image by using the physicalcharacteristic parameter acquired by the physical-characteristicparameter acquisition unit; and a plurality of output units thatrespectively output a plurality of pieces of information about theobject whose sense of quality has been controlled by thesense-of-quality control unit.

The image processing apparatus can further include a functioninformation acquisition unit that acquires function information of theplurality of output units, in which the sense-of-quality control unitcontrols the sense of quality of the object in the image by using thephysical characteristic parameter and the function information acquiredby the function information acquisition unit.

The plurality of output units are constituted by display units of a sametype that output images about the object.

The plurality of output units output the images about the object at asame pixel position.

The plurality of output units are constituted by at least one displayunit that outputs an image about the object, and an output unit thatoutputs information other than the image about the object.

The physical characteristic parameter can be reflection characteristicinformation indicating a reflection characteristic of the object, andthe output unit can output the reflection characteristic of the object.

One output unit can output a specular reflection component of thereflection characteristic of the object, and another output unit canoutput a diffuse reflection component of the reflection characteristicof the object.

The one output unit can output the specular reflection component of thereflection characteristic of the object to a front, and the other outputunit can output the diffuse reflection component of the reflectioncharacteristic of the object to a back.

The one output unit can be a device having a high peak luminance, andthe other output unit can be a device having a low peak luminance.

The physical characteristic parameter can be a wavelength of the object,and the output unit can perform output with respect to a wavelength ofthe object.

The one output unit can be an electronic paper and outputs an objectcolor of the wavelength of the object, and the other output unit can bean LCD (liquid crystal display) and outputs a light source color of thewavelength of the object.

The physical characteristic parameter can be a frequency component ofthe object, and the output unit can perform output with respect to thefrequency component of the object.

The physical characteristic parameter can be the frequency component ofthe object, and the output unit can perform output with respect to atexture of the frequency component of the object.

The one output unit can be a tactile display and outputs the texture ofthe frequency component of the object, and the other output unit can bean LCD (liquid crystal display) and outputs a structure of the frequencycomponent of the object.

The physical characteristic parameter can be a depth or deepness of theobject, and the output unit can perform output with respect to the depthor deepness of the object.

The physical characteristic parameter can be information on a timedirection of the object, and the output unit can perform output withrespect to a mobile object and a stationary object in the time directionof the object.

An image processing method according to an aspect of the presentdisclosure includes: by an image processing apparatus, acquiring aphysical characteristic parameter regarding an object of an image;controlling a sense of quality of the object in the image by using theacquired physical characteristic parameter; and respectively outputtinga plurality of pieces of information about the object whose sense ofquality has been controlled, to a plurality of display units.

In an aspect of the present disclosure, a physical characteristicparameter regarding an object of an image is acquired and a sense ofquality of the object in the image is controlled by using the acquiredphysical characteristic parameter. Then, a plurality of pieces ofinformation about the object whose sense of quality has been controlledare respectively output to a plurality of display units.

Advantageous Effects of Invention

In accordance with an aspect of the present disclosure, images can beprocessed. In particular, it is possible to more favorably reproduce asense of quality as if there were a real object.

Note that the effects described in the present specification are merelyillustrative, the effects of the present technology are not limited tothe effects described in the present specification, and additionaleffects may be provided.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 A diagram describing a concept of the present technology.

FIG. 2 A diagram describing a sense-of-quality control method accordingto the present technology.

FIG. 3 A block diagram showing a main configuration example of an imageprocessing apparatus.

FIG. 4 A block diagram showing another configuration example of theimage processing apparatus.

FIG. 5 A diagram showing an example of a display system.

FIG. 6 A diagram describing an example of processing of the presenttechnology.

FIG. 7 A diagram describing another example of the processing of thepresent technology.

FIG. 8 A flowchart describing image processing of the image processingapparatus of the present technology.

FIG. 9 A flowchart describing measurement/estimation/integrationprocessing.

FIG. 10 A flowchart describing processing based on viewing-environmentinformation.

FIG. 11 A flowchart describing reflection characteristic adjustmentprocessing based on a material of an object.

FIG. 12 A flowchart describing processing based on line-of-sightinformation.

FIG. 13 A flowchart describing reflection characteristic adjustmentprocessing based on operation information.

FIG. 14 A flowchart describing output control processing based on deviceinformation.

FIG. 15 A flowchart describing output control processing based on deviceinformation.

FIG. 16 A flowchart describing output control processing based on deviceinformation.

FIG. 17 A flowchart describing output control processing based on deviceinformation.

FIG. 18 A block diagram showing a main configuration example of apersonal computer.

MODE(S) FOR CARRYING OUT THE INVENTION

Hereinafter, modes for carrying out the present disclosure (hereinafter,referred to as embodiments) will be described. Note that descriptionswill be made in the following order.

-   1. Outline of Present Technology-   2. Configuration Example-   3. Embodiment-   4. Processing Example-   5. Configuration Example of Computer

<1. Outline of Present Technology>

[Concept of Present Technology]

A concept of the present technology will be described with reference toFIG. 1. The present technology is for improving a sense of quality of anobject in an image.

In the real world, those that a person feels with eyes, that is,physical information (material information) indicating how light entersthe eyes is fundamentally necessary information. This physicalinformation is, for example, solar light (illumination: lighting), ashape (geometry) of an object, a reflection component (reflectance) ofthe object, and the like. With this physical information, light can betheoretically reproduced.

This physical information enters sensors of a camera. Therefore, thisphysical information can be estimated and acquired as characteristicinformation of the object (region information and recognitioninformation of the object) on the basis of information (imagestatistical amount, diffuse/specular reflection component, illuminationinformation at photographing time) extracted from an image (2D image).

Note that processing of extracting information from an image isprocessing considering characteristics of light and processing ofestimating it as the characteristic information of the object isobject-based processing.

Therefore, in the present technology, information extracted from animage is utilized as the characteristic information of the object tocontrol the sense of quality of (a surface of) the object in the image.Further, it is also possible to utilize physical information, which ismeasured and acquired from the real world, in such control.

In addition, the sense of quality of the object in the image isassociated with not only this physical information but also, actually,information on an environment in which viewing is performed(illumination information of viewing environment) and informationindicating how a person feels (perception: hyperacuity). Therefore, inthe present technology, the sense of quality of the object in the imageis controlled and an image is remade by not only utilizing this physicalinformation but also utilizing the information on the environment inwhich viewing is performed (illumination information of viewingenvironment) and the information indicating how the person feels.

In this manner, in the present technology, the sense of quality of theobject in the image is controlled. Here, in the present specification,the sense of quality refers to psychological feelings (psychologicalfactors) of a person, which is caused with respect to properties(physical factors) of a material.

That is, in the present specification, the sense of quality is definedas one including parameters representing physical characteristics of theobject that are the physical factors and parameters representingcognitive sensitivity to the object that are the psychological factors.

Therefore, the sense-of-quality control in the present specificationmeans control on the parameters of those physical factors andpsychological factors. Note that, although the description “physicalcharacteristic parameters that are the parameters of the physicalfactors are controlled in the sense-of-quality control” will be merelymade hereinafter, the parameters representing the cognitive sensitivityto the object that are the parameters of the psychological factors areactually controlled also in such a case.

In addition, in the present technology, corresponding not only to animage output but also to various output devices, the sense-of-qualitycontrol is performed not only on an output image but also on additionalinformation associated with the output image by analyzing information onthe output devices.

With this, the result of analyzing the characteristic information of theobject can be output as information other than the image. Therefore,more realistic visual expression becomes possible.

Further, in the present technology, corresponding to the plurality ofoutput devices, the sense-of-quality control is also performed on aplurality of output images having different features by analyzing theinformation on the output device.

With this, it is compatible with various displays such as a light-fielddisplay and a hybrid display like a DFD (Depth-fused Display).Therefore, more realistic visual expression becomes possible.

Note that, by inputting these output image and additional informationinto an output device, it becomes possible to use the additionalinformation for device control.

Further, information on user's operations performed during display ofthe image, line-of-sight information, and viewing information areoperation information regarding user's operations performed duringdisplay of the image. Therefore, hereinafter, they will also becollectively referred to as user operation information.

[Sense-of-Quality Control Method According to Present Technology]

Next, a sense-of-quality control method according to the presenttechnology will be described with reference to FIG. 2.

First of all, an object of the real world is photographed and an imageof the object is input into a measurement and estimation block 1. In themeasurement and estimation block 1, physical characteristic parametersindicating the physical characteristics of the object are measured andacquired from the real world at the photographing time of the object.Alternatively, in the measurement and estimation block 1, the physicalcharacteristic parameters indicating the physical characteristics of theobject are estimated and acquired from the input image of the object.For example, the above-mentioned illumination and structure andreflection characteristic of the object are acquired as the physicalcharacteristic parameters.

The acquired physical characteristic parameters of the object aremodeled in a real-world modeling block 2. The modeled physicalcharacteristic parameters of the object are input into asense-of-quality control block 3.

In the sense-of-quality control block 3, the sense of quality of theobject in the image is controlled in a manner that depends on themodeled physical characteristic parameters of the object and an amountof feature (texture) obtained from the image. As an example of thesense-of-quality control, the physical characteristic parameters arechanged to make reflection easy, for example. With this, the opticalcharacteristics of the object are optimized. Further, for example, ifthere is a portion insufficient in texture, it is suitably recovered.That is, in the sense-of-quality control block 3, as thesense-of-quality control, the physical characteristic parameters relatedto the shininess and the transparency of the appearance are changed(controlled) to increase them.

Further, in the sense-of-quality control block 3, the user operationinformation regarding user's operations performed during display of theimage (e.g., operation information, line-of-sight information, andviewing information) is acquired and analyzed and used forsense-of-quality control thereof. In addition, information on an outputdevice other than the image output and a plurality of image outputdevices having different features is analyzed and used forsense-of-quality control.

In a rendering and retouch block 4, the image is recomposed (rendered)and one or more images that are the result of finely adjusting the imagequality are output (the number of images depends on the number of imageoutput devices) in order to reconfigure the image in accordance with theresult (changed parameters) of controlling the sense of quality.Further, in accordance with the sense-of-quality control, the additionalinformation that is the information other than the image, which isassociated with the image, is generated. In accordance with thegenerated additional information, it is output to the output deviceother than the image output.

By the above-mentioned processing, in accordance with the presenttechnology, optimization of illumination light, enhancement of theshininess, or reproduction of the transparency in the image, forexample, are performed if the input image is different from the actualappearance. That is, a situation when the object is actually seen can bevisually reproduced.

Further, in accordance with the present technology, the image qualityand the display method are changed in a manner that depends on the useroperation information (user's interaction). Therefore, feelings otherthan the appearance such as the sense of touch of the object can also begiven.

In addition, in accordance with the present technology, correspondingnot only to the image output but also to various output devices, theinformation on the output devices is analyzed, and the sense-of-qualitycontrol is performed not only on the output image but also on theadditional information associated with the output image. Further, inaccordance with the present technology, the sense-of-quality control isperformed not only on one image output but also on a plurality of imageoutputs having different features. With this, more realistic visualexpression becomes possible.

<2. Configuration Example of Apparatus>

[Configuration Example of Image Processing Apparatus]

FIG. 3 is a block diagram showing a configuration of an embodiment of animage processing apparatus to which the present disclosure is applied.

An image processing apparatus 11 shown in FIG. 3 acquires physicalcharacteristic parameters regarding an object in an image input afterthe object of the real world is photographed as described above, andacquires operation information of a user (viewer). Further, the imageprocessing apparatus 11 includes not only an image output but alsovarious output devices. The image processing apparatus 11 also analyzesinformation on those output devices, controls the sense of quality ofthe object in the image in a manner that depends on the physicalcharacteristic parameters and the operation information and the analysisresults of those output devices, and outputs the image whose sense ofquality of the object has been controlled and the additional informationassociated with the image.

The image processing apparatus 11 is configured to include aphotographing-environment information acquisition unit 21, aphotographing-environment information analyzer 22, an objectcharacteristic analyzer 23, a characteristic information integrationunit 24, a viewing-environment information analyzer 25, a characteristiccontrol unit 26, an image composition unit 27, and an operationalenvironment information analyzer 28. In addition, the image processingapparatus 11 is configured to include a prior-knowledge database 29, anadditional-information generation unit 30, a device 31-1, a device 31-2,and a device information analyzer 33.

The photographing-environment information acquisition unit 21, thephotographing-environment information analyzer 22, and the objectcharacteristic analyzer 23 correspond to the measurement and estimationblock 1 of FIG. 2. The characteristic information integration unit 24corresponds to the real-world modeling block 2 of FIG. 2. Thecharacteristic control unit 26 corresponds to the sense-of-qualitycontrol block 3 of FIG. 2. The image composition unit 27 corresponds tothe rendering and retouch block 4.

The photographing-environment information acquisition unit 21 takes animage of an object, inputs the image of the object, and supplies theinput image, which has been input, to the object characteristic analyzer23 and the image composition unit 27. Further, thephotographing-environment information acquisition unit 21 acquiresphotographing-time information on the environment and the object at thephotographing time of the image of the object and supplies the acquiredphotographing-time information to the photographing-environmentinformation analyzer 22.

The photographing-environment information analyzer 22 analyzes thephotographing-time information acquired at the photographing time fromthe photographing-environment information acquisition unit 21. Thephotographing-environment information analyzer 22 supplies the analyzedphotographing-time information to the characteristic informationintegration unit 24.

The object characteristic analyzer 23 estimates and analyzescharacteristics of the object on the basis of the input image from thephotographing-environment information acquisition unit 21. The objectcharacteristic analyzer 23 supplies the analyzed image estimationinformation to the characteristic information integration unit 24. Inthe object characteristic analyzer 23, information on details (e.g.,portions having high sampling rate) that could not be acquired at thephotographing time as the photographing-time information can be acquiredby estimating it on the basis of the image.

The characteristic information integration unit 24 integrates thephotographing-time information from the photographing-environmentinformation analyzer 22 and the image estimation information from theobject characteristic analyzer 23. The characteristic informationintegration unit 24 supplies it to the characteristic control unit 26and the prior-knowledge database 29 as the physical characteristicparameter regarding the object.

The viewing-environment information analyzer 25 acquires and analyzesinformation on a viewing environment that is an environment in viewingan image (viewing-environment information). The viewing-environmentinformation analyzer 25 supplies the analyzed viewing-environmentinformation to the characteristic control unit 26 as aviewing-environment parameter.

The characteristic control unit 26 uses the physical characteristicparameters regarding the object from the characteristic informationintegration unit 24, the viewing-environment parameter from theviewing-environment information analyzer 25, operational environmentparameter from the operational environment information analyzer 28, anddevice information from the device information analyzer 33, as controlparameters. Specifically, for example, the characteristic control unit26 determines processing contents for controlling the sense of quality,for example, changing the image quality, adjusting the reflectioncharacteristic, or changing the shape, on the basis of the physicalcharacteristic parameters and the device information. The characteristiccontrol unit 26 supplies information on the determined processingcontents to the image composition unit 27 and the additional-informationgeneration unit 30 and causes the image composition unit 27 and theadditional-information generation unit 30 to perform thesense-of-quality control. Further, the characteristic control unit 26causes the image composition unit 27 and to perform the sense-of-qualitycontrol in a manner that depends on the viewing-environment parameter,the operational environment parameter, and the device information foroptimizing the sense-of-quality control. That is, in the imageprocessing apparatus 11, more realistic visual expression becomespossible due to an additional output other than the image from thedevice 31-2 besides the image output of the device 31-1.

Under the control of the characteristic control unit 26, the imagecomposition unit 27 recomposes (renders) and adjusts the input imagefrom the photographing-environment information acquisition unit 21 andoutputs a recomposition result to the device 31-1 as the output image.

The operational environment information analyzer 28 acquires andanalyzes the operational environment information of the user withrespect to the image, which is performed during display of the image.

Note that examples of the operational environment information caninclude line-of-sight information of the viewer, operation informationusing an operation unit such as a touch panel and a mouse, sensorinformation given to the display, and viewing time information of theuser. Examples of the line-of-sight information of the viewer caninclude information on position and range of a gazed region andinformation on the number of eye blinks. Examples of the operationinformation can include information on a touch position, a pointerposition, a flick range, how to move arms which is obtained by gesturerecognition, and the like. Examples of the sensor information caninclude information on a tilt of the display and a moving speed thereof.

The operational environment information analyzer 28 supplies theanalyzed operational environment information to the characteristiccontrol unit 26 as the operational environment parameter.

The prior-knowledge database 29 retains material information of theobject, such as a jewel, a metal portion, water, leather, and theinformation other than the image, which regards size, hardness, and thelike of the object. For example, in the prior-knowledge database 29, onthe basis of the physical characteristic parameters regarding the objectsupplied from the characteristic information integration unit 24, thedesk, jewel, metal, or the like is searched for and a sense of touch,sound information, and the like of the searched desk, jewel, or metalare supplied to the additional-information generation unit 30. That is,the information of this prior-knowledge database 29 is used, for exampleif the information on the device is not acquired in the deviceinformation analyzer 33 or if the information on the device is notanalyzed in generating the additional information.

The additional-information generation unit 30 generates, on the basis ofinformation on the determined processing contents (sense-of-qualitycontrol), additional information that is, for example, additionalinformation of sound, vibration, a sense of touch, and the like of theobject of the image and is for controlling the device 31-2. Theadditional-information generation unit 30 supplies the generatedadditional information to the device 31-2. That is, theadditional-information generation unit 30 outputs the result ofanalyzing the characteristic information of the object as additionalinformation other than the image. Examples of the additional informationcan include sound and sense-of-touch information expressing the sense ofquality of the object, information on the specular reflection component,gamma characteristic information, and color information.

The device 31-1 is, for example, constituted by an LCD (liquid crystaldisplay) and the like and displays the output image from the imagecomposition unit 27. Further, the device 31-1 is, for example, an LCDitself and supplies information on a function and a performance as adevice for displaying an image, for example, to the device informationanalyzer 33.

The device 31-2 is constituted by a device that outputs additionalinformation that is additional information such as sound, vibration, anda sense of touch regarding the object of the image, other than theimage. For example, the device 31-2 is constituted by a speaker thatoutputs sound and a device that is stacked on the device 31-1 andprovides the user with a sense of touch by outputting the air,vibration, temperature, or the like in a user's touch operation.

The device information analyzer 33 acquires and analyzes information onthe performance and the function of the devices from the device 31-1 andthe device 31-2. The device information analyzer 33 supplies theanalyzed information of the devices to the characteristic control unit26.

Note that the control parameters are constituted by physicalcharacteristic parameters regarding the object in which thephotographing-time information from the photographing-environmentinformation analyzer 22 and the image estimation information from theobject characteristic analyzer 23 are integrated and theviewing-environment parameter from the viewing-environment informationanalyzer 25.

The physical characteristic parameters regarding the object isconfigured to include deepness and shape information of the object,illumination information at the photographing time, material informationof the object, and reflection characteristic information of the object.Note that the illumination information at the photographing time alsoinfluences the color and the like of the object, and it can be said thatit is information associated with the object or associated with thebackground of the object. Therefore, it is included in the physicalcharacteristic parameters regarding the object.

The viewing-environment parameter is configured to includeviewing-environment light information.

[Another Configuration Example of Image Processing Apparatus]

FIG. 4 shows another configuration of the embodiment of the imageprocessing apparatus to which the present disclosure is applied. In theexample of FIG. 4, an image processing apparatus 51 controls the senseof quality of the object in the image not only with respect to one imageoutput but also with respect to a plurality of image outputs havingdifferent features and outputs the image whose sense of quality of theobject has been controlled. Therefore, the input image is not limited toa single image and multi-viewpoint images may be input as the inputimages.

The image processing apparatus 51 shown in FIG. 4 is common to the imageprocessing apparatus 11 of FIG. 3 in that it includes thephotographing-environment information acquisition unit 21, thephotographing-environment information analyzer 22, the objectcharacteristic analyzer 23, the characteristic information integrationunit 24, the viewing-environment information analyzer 25, thecharacteristic control unit 26, and the device information analyzer 33.

The image processing apparatus 51 is different from the image processingapparatus 11 of FIG. 3 in that the prior-knowledge database 29 and theadditional-information generation unit 30 are excluded and the imagecomposition units 27-1 to 27-2 and the devices 31-1 to 31-2 are changedinto image composition units 27-1 to 27-N and devices 31-1 to 31-N(N>1).

That is, the devices 31-1 to 31-N are, for example, constituted byrespective displays (or pixels) constituting a display system such as ahybrid display of a DFD (Depth-fused Display: multi-layer display) shownin A of FIG. 5, multiple projectors shown in B of FIG. 5, a light-fielddisplay shown in C of FIG. 5, or the like. Note that the devices 31-1 to31-N may be devices of the same type or may be devices of differenttypes.

The multi-layer display includes displays in the front and rear as shownin A of FIG. 5. As fusion in a deepness direction, a 3D image can beperceived by displaying objects arranged in a mutually overlappingmanner in the front and rear. Further, as continuous deepnessexpression, it is possible to perceive a high-luminance object in thefront as if it were positioned on a front side and a low-luminanceobject in the front as if it were positioned on a deep side, bydisplaying, in the front, objects whose luminance ratio is changed atthree steps from dark to bright and displaying, in the rear, objectswhose luminance ratio is changed at three steps from bright to dark inthe entire screen in an overlapping manner.

In the multiple projectors, as shown in B of FIG. 5, a plurality ofprojectors project light onto a single display surface. It is sensedwith a camera or the like. The sensed information is subjected toanalysis & signal processing and fed back to each projector. A video isthus adjusted.

The light field refers to a method of handling all rays present in aspace. It considers even directions in which rays travel. A normalcamera obtains ray intensity in a two-dimensional plane. On the otherhand, a camera adapted for the light field is capable offour-dimensionally (in principle, five-dimensionally) storing even inwhich directions rays have passed at a certain position. The light-fielddisplay is a display capable of controlling (reproducing and realizing)the “directions of rays”.

For example, in a case of the light-field display, ray information aswell as luminance information differ between respective displays (orpixels) constituting the light-field display. For example, in a case ofthe DFD, the deepness information as well as the luminance informationdiffer between respective displays constituting the DFD.

Therefore, the characteristic control unit 26 determines the processingcontents of each display for controlling the sense of quality as thedisplay system on the basis of the physical characteristic parametersfrom the characteristic information integration unit 24, information(e.g., position information, deepness information, and ray information)from each of the devices 31-1 to 31-N from the device informationanalyzer 33, and the like. Then, the characteristic control unit 26causes the image composition units 27-1 to 27-N to perform thedetermined sense-of-quality control.

Under the control of the characteristic control unit 26, the imagecomposition units 27-1 to 27-N recompose (render) and adjust the inputimage from the photographing-environment information acquisition unit 21and respectively output the recomposition result to the respectivelycorresponding devices 31-1 to 31-N as the output image.

The devices 31-1 to 31-N are constituted by respective displaysconstituting a display system such as a hybrid display like a DFD(Depth-fused Display), representing the example described above withreference to FIG. 5, for example. The devices 31-1 to 31-N display theoutput images from the corresponding image composition units 27-1 to27-N. Further, the devices 31, for example, supply individual deviceinformation such as their own function and performance information,position information, ray information, and direction information,deepness information to the device information analyzer 33.

The device information analyzer 33 analyzes device information from thedevices 31-1 to 31-N and supplies to the characteristic control unit 26.

<3. Embodiment>

Next, a processing example in the image processing apparatus 51 will bedescribed with reference to FIG. 6. In the example of FIG. 6, as theprocessing example, output targets, a plurality of information outputsoutput from the output targets, and effects corresponding thereto areshown.

For example, if the output target (devices 31-1 to 31-N) is amulti-layer display, the plurality of information outputs can beclassified into an image of an object on a front side and an image of anobject on a deep side. As an effect thereof, emphasis of thestereoscopic sense can be obtained. Further, the plurality ofinformation outputs can be classified into outputs of a diffusereflection component and a specular reflection component. As an effectthereof, the shininess utilizing a dynamic range can be enhanced.Further, the plurality of information outputs can be classified into anobject side and a light source color. As an effect thereof, a reductionin power consumption can be achieved by displaying an invariable objectcolor on the electronic paper and displaying a difference on the LCD.

If the output target is multiple projectors, the plurality ofinformation outputs can be classified into the diffuse reflectioncomponent and the specular reflection component. As an effect thereof,an improvement in the shininess utilizing the dynamic range can beachieved.

If the output target is an LCD or an OLED (organic electroluminescence),the plurality of information outputs can be classified into the diffusereflection component and the specular reflection component. As an effectthereof, power consumption can be reduced and screen burn-in can beprevented while keeping visibility due to lowing of the specularreflection component.

If the output target is a tactile display, the plurality of informationoutputs can be classified into a structure component and a texturecomponent. As an effect thereof, the sense of touch utilizing thetexture component can be reproduced.

If the output target is video+audio, the plurality of informationoutputs can be classified into a video and audio corresponding to amaterial. As an effect thereof, audio corresponding to the object can beoutput. The plurality of information outputs can be classified intoluminance information and color information (wavelength). Further, itcan be classified into being focused and being unfocused and can beclassified into a high-frequency component and a low-frequencycomponent.

Further, FIG. 7 is a diagram describing another processing example. Inthe example of FIG. 7, as the processing example, characteristics of anobject to be used, types of a plurality of devices (devices 31-1 to31-N) (same type or different types) and positions of pixels thereof(same position or pixel shift), and effects thereof are shown.

As the characteristic of the object, luminance/color of the wavelengthis used in the plurality of projectors at the same position (i.e., sametype). The effect of enhancing the color reproducibility can be thusobtained.

As the characteristic of the object, the light source color/object colorof the wavelength is used in the plurality of projectors at the sameposition (i.e., same type). The effect of enhancing the colorreproducibility can be thus obtained. Further, the light sourcecolor/object color of the wavelength is utilized as the characteristicof the object, and the object color is displayed on the electronic paperand the light source color is displayed on the LCD at the same position.The effect of reducing the power consumption can be thus obtained. Notethat details of this processing example will be described later withreference to FIG. 16 as a processing example e3.

As the characteristic of the object, the frequency component is used inthe plurality of projectors (of the same type) at the same position foreach frequency component. The effect of enhancing the color reproductioncan be thus obtained. The effect of enhancing the resolution can beobtained by using the frequency component as the characteristic of theobject for each frequency component in a plurality of pixel-shiftprojectors (of the same type).

As the characteristic of the object, for each frequency component, a lowfrequency is used in a low-resolution device and a high frequency isused in a high-resolution device at the same position or with pixelshift. The effect of enhancing the sense of resolution can be thusobtained.

As the characteristic of the object, the texture, structure (shape) ofthe frequency component is utilized, and the texture is displayed on theentire surface in the LCD or OLED of the same type at the same position.The effect of emphasizing the stereoscopic sense can be thus obtained.Further, the sense of touch can be reproduced by utilizing the texture,structure (shape) of the frequency component as the characteristic ofthe object and using the texture on the tactile display and thestructure on the LCD or OLED at the same position. Note that details ofthis processing example will be described later with reference to FIG.15 as a processing example e2.

As the characteristic of the object, the focus of the frequencycomponent is utilized, and an unfocused portion is used in the back anda focused portion is used in the front at the same position. The effectof emphasizing the stereoscopic sense can be thus obtained.

As the characteristic of the object, the focus of the frequencycomponent is utilized, and an unfocused portion is used in thelow-resolution device and a focused portion is used in thehigh-resolution device at the same position. The effect of enhancing thesense of resolution can be thus obtained.

As the characteristic of the object, the exposure of the frequencycomponent is utilized, and exposure-different videos are integrated indevices of the same type at the same position. The effect of enhancingthe dynamic range can be thus obtained.

As the characteristic of the object, the specular reflection/diffusereflection of the reflection component is used in the plurality ofprojectors (of the same type) at the same position. Due to the emphasisof the specular reflection, the effect of enhancing the shininess can bethus obtained. Note that details of this processing example will bedescribed later with reference to FIG. 14 as a processing example e1.

As the characteristic of the object, the specular reflection of thereflection component is displayed in the front and the diffusereflection displayed in the back at the same position. The effects ofenhancing the shininess and emphasizing the stereoscopic sense can beobtained.

As the characteristic of the object, the specular reflection/diffusereflection of the reflection component is utilized, and the specularreflection is suppressed in the LCD or OLED at the same position. Theeffects of reducing the power consumption and preventing screen burn-incan be thus obtained.

As the characteristic of the object, the specular reflection/diffusereflection of the reflection component is utilized, and display isperformed using different degrees of emphasizing the specular reflectionand parallax between L and R of an HMD (head-mounted display) at thesame position. The effect of enhancing the shininess can be thusobtained.

As the characteristic of the object, the specular reflection of thereflection component is displayed on a device having a high peakluminance and the diffuse reflection is displayed on a device having alow peak luminance at the same position. The effect of enhancing thedynamic range can be thus obtained. Note that details of this processingexample will be described later with reference to FIG. 17 as aprocessing example e4.

As the characteristic of the object, interreflection of the reflectioncomponent is utilized, and the interreflection is displayed on atransparent display and the others are displayed on the LCD at the sameposition. The effect of enhancing the transparency can be thus obtained.

As the characteristic of the object, for each depth of thedepth/deepness, the front object is displayed in the front in the LCD orOLED of the same type at the same position. The effect of emphasizingthe stereoscopic sense can be thus obtained.

As the characteristic of the object, for each depth of thedepth/deepness, a background portion is displayed in the low-resolutiondevice and a foreground portion is displayed in the high-resolutiondevice at the same position. The effect of enhancing the sense ofresolution can be thus obtained.

As the characteristic of the object, the object of interest of thedepth/deepness is utilized, and the object of interest is displayed inthe front in the LCD or OLED of the same type at the same position. Theeffect of emphasizing the stereoscopic sense can be thus obtained.

As the characteristic of the object, the object of interest of thedepth/deepness is utilized, and the object of interest is displayed on afront mobile display and the background is displayed on a large-screendisplay at the same position. The effect of emphasizing the stereoscopicsense can be thus obtained.

As the characteristic of the object, a mobile object and a stationaryobject in the time direction are utilized, and the stationary object isdisplayed on the electronic paper and the mobile object is displayed onthe LCD at the same position. The effect of reducing the powerconsumption can be thus obtained.

As the characteristic of the object, other audio information,sense-of-taste information, sense-of-smell information, and meaning(text) are utilized, and a video is displayed on the display, audioassociated with the object is displayed to the speaker and to a devicethat outputs visual assistance at the same position. More realisticvisual expression can be thus realized.

<4. Processing Example>

[Example of Image Processing]

Next, image processing of will be described with reference to theflowchart of FIG. 8.

In Step S11, the photographing-environment information acquisition unit21 takes an image of an object and inputs the image of the object. Thephotographing-environment information acquisition unit 21 supplies theinput image, which has been input, to the object characteristic analyzer23 and the image composition unit 27.

In Step S12, the photographing-environment information acquisition unit21, the photographing-environment information analyzer 22, and theobject characteristic analyzer 23 performsmeasurement/estimation/integration processing. Details of thismeasurement/estimation/integration processing will be described laterwith reference to FIG. 9. Through the processing of Step S12, themeasured physical characteristic parameters and the estimated physicalcharacteristic parameters (the reflection characteristic of the object,photographing-environment light, an object shape, and a material of theobject, etc.) are supplied to the characteristic control unit 26.

In Step S13, the viewing-environment information analyzer 25 and thecharacteristic control unit 26 perform processing based onviewing-environment information that is information on an environment inviewing an image. Details of this processing based on theviewing-environment information will be described later with referenceto FIG. 10. Through the processing of Step S13, the processing contentsof the sense-of-quality control based on the viewing-environmentinformation are determined.

In Step S14, the characteristic control unit 26 performs reflectioncharacteristic adjustment processing based on the material of theobject. This reflection characteristic adjustment processing will bedescribed later with reference to FIG. 11. Note that, at this time,integrated physical characteristic parameters (material and reflectioncharacteristic) is referred to by the characteristic informationintegration unit 24. Through the processing of Step S14, the processingcontents of the sense-of-quality control based on the reflectioncharacteristic are determined.

In Step S15, the operational environment information analyzer 28 and thecharacteristic control unit 26 perform processing based on line-of-sightinformation included in the user operation information. This processingbased on the line-of-sight information will be described later withreference to FIG. 12. Through the processing of Step S15, the processingcontents of the sense-of-quality control based on the line-of-sightinformation are determined.

In Step S16, the operational environment information analyzer 28 and thecharacteristic control unit 26 perform processing based on operationinformation included in the user operation information. The processingbased on this operation information will be described later withreference to FIG. 13. Through the processing of Step S16, the processingcontents of the sense-of-quality control based on the operationinformation are determined.

In Step S17, the characteristic control unit 26 and the deviceinformation analyzer 33 perform output control processing based on thedevice information. The output control processing based on this deviceinformation will be described later with reference to FIGS. 14 to 17.

Due to the output control of Step S17, the sense of quality iscontrolled in a manner that depends on the device information and theimage of the object whose sense of quality has been controlled is outputto the devices 31-1 to 31-N (N>1). Note that, in a case of the imageprocessing apparatus 11 of FIG. 3, due to the output control of StepS17, the image of the object whose sense of quality has been controlledis output from the device 31-1 and the additional information associatedwith the image is output from the device 31-2.

As described above, all physical characteristic parameters obtained atthe photographing time or from the image are used. Then, in addition, aslong as these physical characteristic parameters are accuratelyacquired, optimization of illumination light, enhancement of theshininess, and reproduction of the transparency in the image areperformed by changing them even if the reproducibility of the inputimage is poor. That is, a situation when the object is actually seen canbe visually reproduced like CG (computer graphics).

In the above-mentioned manner, in the present technology, the videoquality can be improved by measuring the physical characteristicparameters regarding the object (shape, reflection characteristic,illumination, etc.) and controlling the characteristics thereof.

Further, a plurality of images and information associated with theimages can be output in a manner that depends on the device information.Therefore, it is possible to more favorably reproduce a sense of qualityas if there were a real object. For example, the deepness sense and thelike can be provided by the display of the DFD and the like or feelingsother than the appearance, such as the sense of touch, can also begiven.

Next, the measurement/estimation/integration processing of Step S12 ofFIG. 8 will be described with reference to the flowchart of FIG. 9.

In Step S31, the photographing-environment information acquisition unit21 measures photographing-environment light and supplies the measuredphotographing-environment light to the photographing-environmentinformation analyzer 22. The photographing-environment informationanalyzer 22 analyzes the photographing-environment light from thephotographing-environment information acquisition unit 21 and suppliesthe analyzed result, the information on the photographing-environmentlight to the characteristic information integration unit 24.

In Step S32, the photographing-environment information acquisition unit21 measures a reflection characteristic of the object and supplies themeasured reflection characteristic and material of the object to thephotographing-environment information analyzer 22. Thephotographing-environment information analyzer 22 analyzes thereflection characteristic and material of the object from thephotographing-environment information acquisition unit 21 and suppliesthe analyzed reflection characteristic and material of the object to thecharacteristic information integration unit 24.

In Step S33, the photographing-environment information acquisition unit21 measures an object shape and supplies the measured object shape tothe photographing-environment information analyzer 22. Thephotographing-environment information analyzer 22 analyzes the objectshape from the photographing-environment information acquisition unit 21and supplies the analyzed object shape to the characteristic informationintegration unit 24.

In Step S34, the object characteristic analyzer 23 estimates andanalyzes photographing-environment light from the supplied input imageand acquires the analyzed result, the information on thephotographing-environment light. The object characteristic analyzer 23supplies the information on the photographing-environment light to thecharacteristic information integration unit 24.

In Step S35, the characteristic information integration unit 24integrates information on the measured photographing-environment lightand information on the estimated photographing-environment light. Thecharacteristic information integration unit 24 supplies the integratedinformation on the photographing-environment light to the characteristiccontrol unit 26 as the physical characteristic parameter.

In Step S36, the characteristic control unit 26 causes the imagecomposition unit 27 to adjust the white balance of the image withillumination color obtained from the information on thephotographing-environment light.

In Step S37, the object characteristic analyzer 23 estimates andanalyzes reflection characteristic and material of the object andsupplies to the characteristic information integration unit 24information on the analyzed reflection characteristic and material.

In Step S38, the characteristic information integration unit 24integrates information on the measured reflection characteristic of theobject and information on the estimated reflection characteristic of theobject. The characteristic information integration unit 24 supplies theintegrated information on the reflection characteristic of the object tothe characteristic control unit 26 as the physical characteristicparameter.

In Step S39, the characteristic information integration unit 24integrates information on the measured material of the object andinformation on the estimated material of the object. The characteristicinformation integration unit 24 supplies the integrated information onthe material of the object to the characteristic control unit 26 as thephysical characteristic parameter.

In Step S40, the object characteristic analyzer 23 estimates andanalyzes a shape of the object and supplies information on the analyzedshape of the object to the characteristic information integration unit24.

In Step S41, the characteristic information integration unit 24integrates information on the measured shape of object and informationon the estimated shape of the object. The characteristic informationintegration unit 24 supplies the integrated information on the shape ofthe object to the characteristic control unit 26 as the physicalcharacteristic parameter.

In the above-mentioned manner, at the photographing time, the materialof the object and the like as well as illumination light and thereflection characteristic and shape of the object that are thecharacteristics of the object are measured. Further, they are estimatedon the basis of the input image and pieces of information thereon areintegrated on the basis of reliability. The thus integrated physicalcharacteristic parameters is used for sense-of-quality control.Therefore, optimization of illumination light, enhancement of theshininess, and reproduction of the transparency in the image are furtherimproved.

Next, the processing based on the viewing-environment information ofStep S13 of FIG. 8 will be described with reference to the flowchart ofFIG. 10.

In Step S61, the viewing-environment information analyzer 25 acquiresand analyzes information on viewing-environment light, for example, asinformation on a viewing environment that is an environment in viewingan image. The viewing-environment information analyzer 25 suppliesinformation on the analyzed viewing-environment light to thecharacteristic control unit 26 as the viewing-environment parameter.

In the characteristic control unit 26, absolute-luminance information ofthe image at the photographing time is also estimated through the objectcharacteristic analyzer 23 and supplied to the characteristic controlunit 26. In Step S62, the characteristic control unit 26 determineswhether or not an absolute luminance of a target pixel at thephotographing time is higher than a predetermined luminance value. InStep S62, if it is determined that the absolute luminance of the targetpixel at the photographing time is higher, the processing proceeds toStep S63.

In Step S63, the characteristic control unit 26 refers to theviewing-environment parameter from the viewing-environment informationanalyzer 25 and determines whether or not the viewing-environment lightis bright above a predetermined value. In Step S63, if it is determinedthat the viewing-environment light is bright, the processing proceeds toStep S64. In this case, it may be looked dark. Therefore, in Step S64,the characteristic control unit 26 controls the image composition unit27 and sets a contrast adjustment value to be relatively high as thesense-of-quality control.

In Step S63, if it is determined that the viewing-environment light isweak, Step S64 is skipped and the processing proceeds to Step S67.

In Step S62, if it is determined that the absolute luminance of thetarget pixel at the photographing time is lower, the processing proceedsto Step S65. In Step S65, the characteristic control unit 26 refers tothe viewing-environment parameter from the viewing-environmentinformation analyzer 25 and determines whether or not theviewing-environment light is bright above a predetermined value.

In Step S65, if it is determined that the viewing-environment light isweak, the processing proceeds to Step S66. In Step S66, thecharacteristic control unit 26 controls the image composition unit 27 toset the contrast adjustment value to be relatively low as thesense-of-quality control.

In Step S65, if it is determined that the viewing-environment light isbright, Step S66 is skipped and the processing proceeds to Step S67.

In Step S67, the characteristic control unit 26 determines whether ornot the adjustment of the reflection characteristic with respect to allthe pixels has been terminated.

In Step S67, if it is determined that the processing has been terminatedwith respect to all the pixels, the processing based on theviewing-environment information ends. In Step S67, if it is determinedthat the processing on all the pixels has not yet been terminated, theprocessing returns to Step S62 and the processing following this step isrepeated.

As described above, the contrast and reflection characteristic of theimage are adjusted on the basis of the viewing environment, and thesense of quality of the image of the object is controlled. With this,optimization of illumination light, the enhancement of the shininess,and the reproduction of the transparency in the image are performed.

Next, the reflection characteristic adjustment processing based on thematerial in Step S14 of FIG. 8 will be described with reference to theflowchart of FIG. 11.

In Step S81, the characteristic control unit 26 determines whether ornot the specular reflection component is large on the basis of theinformation on the reflection characteristic of the object from thecharacteristic information integration unit 24. In Step S81, if it isdetermined that the specular reflection component is large, theprocessing proceeds to Step S82.

In Step S82, the characteristic control unit 26 determines to emphasizethe specular reflection component as the sense-of-quality control. InStep S81, if it is determined that the specular reflection component issmall, the processing proceeds to Step S83.

In Step S83, the characteristic control unit 26 determines whether ornot the diffuse reflection component is large. In Step S83, if it isdetermined that the diffuse reflection component is large, theprocessing proceeds to Step S84. In Step S84, the characteristic controlunit 26 determines to lower the diffuse reflection component as thesense-of-quality control.

In Step S83, if it is determined that the diffuse reflection componentis small, the processing proceeds to Step S85.

In Step S85, the characteristic control unit 26 determines whether ornot it is the material having strong specular reflection on the basis ofinformation on the material of the object which is integrated by thecharacteristic information integration unit 24. In Step S85, if it isdetermined that it is the material having strong specular reflection,the processing proceeds to Step S86. In Step S86, the characteristiccontrol unit 26 determines to emphasize the specular reflectioncomponent as the sense-of-quality control.

In Step S85, if it is determined that it is not the material havingstrong specular reflection, the processing proceeds to Step S87. In StepS87, the characteristic control unit 26 determines to lower the specularreflection component as the sense-of-quality control.

In Step S88, the characteristic control unit 26 determines whether ornot the adjustment of the reflection characteristic with respect to allthe pixels has been terminated.

In Step S88, if it is determined that the adjustment of the reflectioncharacteristic with respect to all the pixels has been terminated, thereflection characteristic adjustment processing ends. In Step S88, if itis determined that the adjustment of the reflection characteristic hasnot yet been terminated with respect to all the pixels, the processingreturns to Step S81 and the processing following this step is repeated.

As described above, in a manner that depends on the material of theobject, the reflection characteristic of the image is adjusted and thesense of quality of the image of the object is controlled. With this,optimization of illumination light, the enhancement of the shininess,and the reproduction of the transparency in the image are performed.

Next, the processing based on the line-of-sight information in Step S15of FIG. 8 will be described with reference to the flowchart of FIG. 12.

In Step S101, the operational environment information analyzer 28acquires and analyzes, for example, object-of-interest information orline-of-sight information as the information on the operationalenvironment. The operational environment information analyzer 28supplies the information on the analyzed operational environment to thecharacteristic control unit 26 as the operational environment parameter.

In Step S102, the characteristic control unit 26 determines whether ornot the target pixel is an object-of-interest region or a line-of-sighttarget region. In Step S102, if it is determined that it is theobject-of-interest region or the line-of-sight target region, theprocessing proceeds to Step S103.

In Step S103, the characteristic control unit 26 controls the imagecomposition unit 27 to set an image-quality adjustment value of theprocessing up to this time (i.e., processing based on theviewing-environment information, material), for example, to berelatively low as the sense-of-quality control.

In Step S102, if it is determined that the target pixel is not theobject-of-interest region or the line-of-sight target region, Step S103is skipped and the processing proceeds to Step S104.

In Step S104, the characteristic control unit 26 determines whether ornot the processing has been terminated with respect to all the pixels.

In Step S104, if it is determined that the processing has beenterminated with respect to all the pixels, the processing based on theline-of-sight information ends. In Step S104, if it is determined thatthe processing on all the pixels has not yet been terminated, theprocessing returns to Step S102 and the processing following this stepis repeated.

As described above, the contrast and reflection characteristic of theimage are adjusted on the basis of the viewing environment, and thesense of quality of the image of the object is controlled. With this,optimization of illumination light, the enhancement of the shininess,and the reproduction of the transparency in the image are performed.

Next, the processing based on the operation information in Step S16 ofFIG. 8 will be described with reference to the flowchart of FIG. 13.

In Step S121, the operational environment information analyzer 28acquires and analyzes, for example, the operation information (flick,etc.) of the user as the information on the operational environment. Theoperational environment information analyzer 28 supplies the informationon the analyzed operational environment to the characteristic controlunit 26 as the operational environment parameter.

In Step S122, the characteristic control unit 26 determines whether ornot the target pixel is a region touched by the user. In Step S122, ifit is determined that it is the region touched by the user, theprocessing proceeds to Step S123.

In Step S123, the characteristic control unit 26 controls the imagecomposition unit 27 to reset the image-quality adjustment value of theprocessing up to this time (i.e., processing based onviewing-environment information, reflection characteristic,line-of-sight information) to be relatively high as the sense-of-qualitycontrol.

In Step S122, if it is determined that the target pixel is not theregion touched by the user, Step S123 is skipped and the processingproceeds to Step S124.

In Step S124, the characteristic control unit 26 determines whether ornot the processing has been terminated with respect to all the pixels.

In Step S124, if it is determined that the processing has beenterminated with respect to all the pixels, the processing based on theline-of-sight information ends. In Step S124, if it is determined thatthe processing on all the pixels has not yet been terminated, theprocessing returns to Step S122 and the processing following this stepis repeated.

As described above, in a manner that depends on the information on theuser's operation out of the operational environment, the reflectioncharacteristic of the image is adjusted. With this, further optimizationof illumination light, further enhancement of the shininess, and furtherreproduction of the transparency in the image are achieved. That is, itis possible to more favorably reproduce a sense of quality as if therewere a real object.

Further, it is possible to provide high-order sense and the like otherthan the appearance, such as the sense of touch of the object byperforming control as described above in a manner that depends on theuser's operation, interaction with respect to a display device asdescribed above.

The sense can be emphasized. Therefore, a more realistic sense can beexpressed even in displaying on a small screen.

Note that this processing can also be executed by determining whether ornot it is a line-of-sight region of the user as the line-of-sightinformation instead of the operation information. That is, processingcan be performed only on a region of interest based on detection of theline of sight or a specified region based on a touch position.Therefore, the specular reflection component can be partiallyemphasized, for example. In contrast, in a case of feeling a too brightvideo, the specular reflection component can be lowered not to providethe too bright video.

In addition, a reduction in power consumption can be achieved bygradually lowering the specular reflection component in long-durationviewing.

Next, the output control processing based on the device information inStep S17 of FIG. 8 will be described with reference to the flowchart ofFIG. 14. Note that the example of FIG. 14 describes the above-mentionedprocessing example e1 with reference to FIG. 7 and is processingperformed by the image processing apparatus 51 of FIG. 4.

In Step S141, the device information analyzer 33 acquires the deviceinformation from the devices 31-1 to 31-N and supplies the acquireddevice information to the characteristic control unit 26.

In Step S142, the characteristic control unit 26 determines whether ornot it is the multiple projectors, on the basis of the deviceinformation from the device information analyzer 33. In Step S142, if itis determined that it is the multiple projectors, the processingproceeds to Step S143.

In Step S143, the characteristic control unit 26 separates theprocessing contents determined in Step S14 of FIG. 8, that is, aspecular reflection component control result and a diffuse reflectioncomponent control result and causes each of the projectors (devices 31-1to 31-N) to output them. That is, in order to cause each of the devices31-1 to 31-N to output them, the characteristic control unit 26 outputsthe processing contents in which the specular reflection componentcontrol result and the diffuse reflection component control result areseparated, to the image composition units 27-1 to 27-N respectivelycorresponding to the devices 31-1 to 31-N. In response to this, theimage composition units 27-1 to 27-N recomposes the input image on thebasis of the determined processing contents of the sense-of-qualitycontrol and causes respectively corresponding devices 31-1 to 31-N tooutput the recomposed image.

On the other hand, in Step S142, if it is determined that it is not themultiple projectors, the processing proceeds to Step S144. That is,since it is not the multiple projectors, only the image composition unit27-1 and the device 31-1 are the output targets.

Therefore, in Step S144, the image composition unit 27-1 recomposes theinput image, on the basis of the processing contents of thesense-of-quality control, which has been determined in Steps S13 to S16of FIG. 8, and causes each corresponding device 31-1 to output therecomposed image.

As described above, if the multiple projectors are provided as theoutput devices, the specular reflection/diffuse reflection of thereflection component are used in the plurality of projectors at the sameposition as the characteristic of the object. Therefore, due to theemphasis of the specular reflection, the effect of enhancing theshininess can be obtained.

Next, another example of the output control processing based on thedevice information in Step S17 of FIG. 8 will be described withreference to the flowchart of FIG. 15. Note that the example of FIG. 15describes the above-mentioned processing example e2 with reference toFIG. 7 and is processing performed by the image processing apparatus 11of FIG. 3.

In Step S161, the device information analyzer 33 acquires the deviceinformation from the devices 31-1 and 31-2 and supplies the acquireddevice information to the characteristic control unit 26.

In Step S162, the characteristic control unit 26 determines whether ornot it is a hybrid display (tactile display+LCD), on the basis of thedevice information from the device information analyzer 33. For example,if the device 31-1 is an LCD display and the device 31-2 is a tactiledisplay, it is determined in Step S162 that it is the hybrid display andthe processing proceeds to Step S163.

In Step S163, the characteristic control unit 26 estimates a texturecomponent and a structure component on the basis of shape information ofthe object out of the physical characteristic parameters from thecharacteristic information integration unit 24.

In Step S164, the characteristic control unit 26 causes the tactiledisplay (device 31-2) to output the texture component estimated in StepS163. That is, in order to cause the tactile display (device 31-2) tooutput it, the characteristic control unit 26 outputs the processingcontents indicative of the texture component to theadditional-information generation unit 30. Corresponding to this, theadditional-information generation unit 30 generates additionalinformation for providing the sense of touch of the texture componentand outputs the generated additional information to the device 31-2.With this, the device 31-2 that is the tactile display is capable ofperforming output corresponding to the sense of touch of the texturecomponent.

Note that if there is little information regarding the sense of touch ofthe texture component in generating the additional information,information may be obtained from the prior-knowledge database 29.

In Step S165, the characteristic control unit 26 causes the LCD display(device 31-1) to output the structure component estimated in Step S163.That is, in order to cause the LCD display (device 31-1) to output it,the characteristic control unit 26 outputs the processing contentsindicative of the structure component to the image composition unit 27.Corresponding to this, the image composition unit 27, the imagecomposition unit 27 recomposes the input image on the basis of thedetermined processing contents of the sense-of-quality control. Withthis, the device 31-1 that is the LCD display is capable of performingoutput corresponding to the structure component.

On the other hand, in Step S162, if it is determined that it is not thehybrid display, the processing proceeds to Step S166. That is, since itis not the hybrid display, only the image composition unit 27 and thedevice 31-1 are the output targets.

Therefore, in Step S166, the image composition unit 27 recomposes theinput image, on the basis of the processing contents of thesense-of-quality control, which has been determined in Steps S13 to S16of FIG. 8, and causes each corresponding device 31-1 to output therecomposed image.

As described above, if the hybrid display formed of the tactile displayand the LCD is provided as the output device, the specularreflection/diffuse reflection of the reflection component are used inthe plurality of projectors at the same position as the characteristicof the object. Therefore, due to the emphasis of the specularreflection, the effect of enhancing the shininess can be obtained.

Next, another example of the output control processing based on thedevice information in Step S17 of FIG. 8 will be described withreference to the flowchart of FIG. 16. Note that the example of FIG. 15describes the above-mentioned processing example e3 with reference toFIG. 7 and is processing performed by the image processing apparatus 11of FIG. 4.

In Step S181, the device information analyzer 33 acquires the deviceinformation from the devices 31-1 and 31-2 and supplies the acquireddevice information to the characteristic control unit 26.

In Step S182, the characteristic control unit 26 determines whether ornot it is a hybrid display (electronic paper+LCD), on the basis of thedevice information from the device information analyzer 33. For example,if the device 31-1 is an LCD display and the device 31-2 is anelectronic paper, it is determined in Step S182 that it is the hybriddisplay and the processing proceeds to Step S183.

In Step S183, the characteristic control unit 26 estimates an objectcolor component/light source color component on the basis ofphotographing-environment light, a reflection characteristic, and shapeinformation of the object out of the physical characteristic parametersfrom the characteristic information integration unit 24.

In Step S184, the characteristic control unit 26 causes the electronicpaper (device 31-2) to output the object color component estimated inStep S183. That is, in order to cause the electronic paper (device 31-2)to output it, the characteristic control unit 26 outputs the processingcontents indicative of the object color component to theadditional-information generation unit 30. Corresponding to this, theadditional-information generation unit 30 generates additionalinformation for outputting the object color component and outputs thegenerated additional information to the device 31-2. With this, thedevice 31-2 that is the electronic paper is capable of outputting theobject color component.

Note that if there is little information regarding the object colorcomponent in generating the additional information, information may beobtained from the prior-knowledge database 29.

In Step S185, the characteristic control unit 26 causes the LCD display(device 31-1) to output the light source color component estimated inStep S183. That is, in order to cause the LCD display (device 31-1) tooutput it, the characteristic control unit 26 outputs the processingcontents indicative of the structure component to the image compositionunit 27. Corresponding to this, the image composition unit 27, the imagecomposition unit 27 recomposes the input image on the basis of thedetermined processing contents of the sense-of-quality control. Withthis, the device 31-1 that is the LCD display is capable of performingoutput corresponding to the structure component.

On the other hand, in Step S182, if it is determined that it is not thehybrid display, the processing proceeds to Step S186. That is, since itis not the hybrid display, only the image composition unit 27 and thedevice 31-1 are the output targets.

Therefore, in Step S186, the image composition unit 27 recomposes theinput image, on the basis of the processing contents of thesense-of-quality control, which has been determined in Steps S13 to S16of FIG. 8, and causes each corresponding device 31-1 to output therecomposed image.

As described above, if the hybrid display formed of the electronic paperand the LCD is provided as the output device, the light sourcecolor/object color of the wavelength is utilized as the characteristicof the object and the object color is displayed on the electronic paperand the light source color is displayed on the LCD at the same position.Therefore, the effect of reducing the power consumption can be obtained.

Next, the output control processing based on the device information inStep S17 of FIG. 8 will be described with reference to the flowchart ofFIG. 17. Note that the example of FIG. 14 describes the above-mentionedprocessing example e4 with reference to FIG. 7 and is processingperformed by the image processing apparatus 51 of FIG. 4.

In Step S201, the device information analyzer 33 acquires the deviceinformation from the devices 31-1 to 31-N and supplies the acquireddevice information to the characteristic control unit 26.

In Step S202, the characteristic control unit 26 determines whether ornot it is the multiple projectors of different types, on the basis ofthe device information from the device information analyzer 33. In StepS202, if it is determined that it is the multiple projectors ofdifferent types, the processing proceeds to Step S203.

In Step S203, the characteristic control unit 26 causes the projector(device 31-1) having a high peak luminance to output the specularreflection component control result of the processing contentsdetermined in Step S14 of FIG. 8. That is, in order to cause the device31-1 to output it, the characteristic control unit 26 outputs theprocessing contents indicative of the specular reflection componentcontrol result to the image composition unit 27-1 corresponding to thedevice 31-1. In response to this, the image composition unit 27-1recomposes the input image on the basis of the determined processingcontents of the sense-of-quality control and causes each correspondingdevice 31-1 to output the recomposed image.

Further, in Step S204, the characteristic control unit 26 causes theprojector (device 31-2) having a low peak luminance to output thediffuse reflection component control result of the processing contentsdetermined in Step S14 of FIG. 8. That is, in order to cause the device31-2 to output it, the characteristic control unit 26 outputs theprocessing contents indicating the diffuse reflection component controlresult, to the image composition unit 27-2 corresponding to the device31-2. In response to this, the image composition unit 27-2 recomposesthe input image on the basis of the determined processing contents ofthe sense-of-quality control and causes each corresponding device 31-2to output the recomposed image.

On the other hand, in Step S202, if it is determined that it is not themultiple projectors of different types, the processing proceeds to StepS205. That is, since it is not the multiple projectors of differenttypes, only the image composition unit 27-1 and the device 31-1 are theoutput targets.

Therefore, in Step S205, the image composition unit 27-1 recomposes theinput image, on the basis of the processing contents of thesense-of-quality control, which has been determined in Steps S13 to S16of FIG. 8, and causes each corresponding device 31-1 to output therecomposed image.

As described above, if the multiple projectors of different types areprovided as the output devices, as the characteristic of the object, thespecular reflection of the reflection component is displayed on a devicehaving a high peak luminance and the diffuse reflection is displayed ona device having a low peak luminance at the same position. Therefore,the effect of enhancing the dynamic range can be obtained.

Note that, although details of the processing will be omitted, also in acase of other processing of FIGS. 6 and 7, the effects shown in FIGS. 6and 7 can be obtained by performing the processing shown in FIGS. 6 and7.

As described above, all the physical characteristic parameters obtainedat the photographing time or from the image are used. Then, in addition,as long as these physical characteristic parameters are accuratelyacquired, optimization of illumination light, the enhancement of theshininess, and the reproduction of the transparency in the image areperformed by changing them even if the reproducibility of the inputimage is poor. That is, a situation when the object is actually seen canbe visually reproduced like CG (computer graphics).

In the above-mentioned manner, in the present technology, the sense ofquality as if there were a real object can be more realisticallyreproduced. In particular, the image quality and the display method arechanged in a manner that depends on the operational environment (user'sinteraction). Therefore, feelings other than the appearance such as thesense of touch of the object can also be given.

Further, the performance or function information of the output device isacquired and an optimal image is recomposed in a manner that depends onthe acquired performance or function of the output device. Therefore,optimal information can be output to the output device. With this, theeffect that cannot be obtained only by viewing can be enhanced and ahigh-order sense can be expressed. As a result, more realistic visualexpression becomes possible.

By also outputting tactile sensor information, a sense of touch can begiven through a user's touch operation. By also outputting audioinformation of the object, feelings other than the appearance can begiven through a user's touch operation or the like. Further, the resultof analyzing the characteristic information of the object can be outputas the information other than the image. For example, it includes soundand tactile information expressing a sense of quality, information on aspecular reflection component or the like, gamma characteristicinformation, and color information.

In addition, a plurality of images having different features are output.Therefore, more realistic visual expression becomes possible. Output tothe light-field display becomes possible and generation of a viewpointdepending on a viewpoint becomes possible. By outputting the image in amanner that depends on the magnitude of the reflection characteristic tothe multi-layer display like the DFD, expression with a strongerstereoscopic sense becomes possible.

By outputting the image in a manner that depends on the magnitude of thereflection characteristic, compatibility between a conventional displayand an HDR display can be obtained. By separating and outputting theimage as the diffuse reflection component and the specular reflectioncomponent, cooperation between the devices can be performed.

In addition, it is possible to perform lift control of the specularreflection component on the OLED or the LCD, a reduction in powerconsumption and prevention of screen burn-in of a multi-function mobilephone, image composition on a screen by displaying each of the specularreflection component and the diffuse reflection component by using twoprojectors, and the like.

Note that the present invention can be applied to an image processingapparatus, a television apparatus, a projector, and the like and to avideo system including them, and the like.

<5. Configuration Example of Computer>

[Personal Computer]

The above-mentioned series of processing may be executed by hardware ormay be executed by software. If the series of processing is executed bysoftware, programs that configure that software are installed into acomputer. Here, the computer includes a computer incorporated indedicated hardware, a general-purpose personal computer capable ofexecuting various functions by installing various programs, and thelike.

FIG. 18 is a block diagram showing a configuration example of hardwareof a personal computer that executes the above-mentioned series ofprocessing in accordance with programs.

In a personal computer 500, a CPU (Central Processing Unit) 501, a ROM(Read Only Memory) 502, and a RAM (Random Access Memory) 503 areconnected to one another through a bus 504.

An input/output interface 505 is further connected to the bus 504. Aninput unit 506, an output unit 507, a storage unit 508, a communicationunit 509, and a drive 510 are connected to the input/output interface505.

The input unit 506 includes a keyboard, a mouse, a microphone, and thelike. The output unit 507 includes a display, a speaker, and the like.The storage unit 508 includes a hard disk, a nonvolatile memory, and thelike. The communication unit 509 includes a network interface, and thelike. The drive 510 drives a removable medium 511 such as a magneticdisk, an optical disc, a magneto-optical disk, and a semiconductormemory.

As described above, in the personal computer 500 to be configured, theCPU 501 loads programs stored in, for example, the storage unit 508 intothe RAM 503 via the input/output interface 505 and the bus 504 andexecutes them. With this, the above-mentioned series of processing isperformed.

The programs executed by the computer (CPU 501) can be stored on theremovable medium 511 and provided. The removable medium 511 is, forexample, a package medium. The package medium includes a magnetic disk(including flexible disk), an optical disc (CD-ROM (Compact Disc-ReadOnly Memory), DVD (Digital Versatile Disc), etc.), a magneto-opticaldisk, a semiconductor memory, and the like. Additionally oralternatively, the programs can be provided via a wired or wirelesstransmission medium such as a local area network, the Internet, anddigital satellite broadcasting.

In the computer, the programs can be installed into the storage unit 508via the input/output interface 505 by the removable medium 511 beingmounted on the drive 510. Further, the programs can be received by thecommunication unit 509 via the wired or wireless transmission medium andinstalled into the storage unit 508. Besides, the programs can beinstalled into the ROM 502 and the storage unit 508 in advance.

Note that the programs executed by the computer may be programs to beprocessed chronologically in the order described in the presentspecification or may be programs to be processed concurrently or atnecessary stages, for example, upon calling.

Further, in the present specification, steps describing the programsstored in a storage medium include, as a matter of course, processing tobe performed chronologically in the order described and also includeprocessing to be concurrently or individually executed withoutnecessarily needing to be processed chronologically.

Further, in the present specification, the system refers to the entireapparatus constituted by a plurality of devices (apparatuses).

Further, the configuration described above as a single apparatus (orprocessor) may be divided and may be configured as a plurality ofapparatuses (or processors). In contrast, the configurations describedabove as a plurality of apparatuses (or processors) may be unified andconfigured as a single apparatus (or processor). Further, as a matter ofcourse, a configuration other than those described above may be added tothe configuration of each apparatus (or each processor). In addition, aslong as the configuration and operation as the entire system aresubstantially the same, a configuration of a part of a certain apparatus(or processor) may be included in a configuration of another apparatus(or another processor). That is, the present technology is not limitedto the above-mentioned embodiments and various modifications can be madewithout departing from the gist of the present technology.

Although favorable embodiments of the present disclosure have beendescribed in detail with reference to the attached drawings, the presentdisclosure is not limited to such an example. It is obvious that aperson with ordinary skill in the art to which the present disclosurepertains can conceive various changed examples or modified exampleswithin the range of the technical ideas described in the scope of claimsand it should be understood that they also fall within the technicalrange of the present disclosure as a matter of course.

Note that the present technology can also take the followingconfigurations.

(1) An image processing apparatus, including:

-   -   a physical-characteristic parameter acquisition unit that        acquires a physical characteristic parameter regarding an object        of an image;

a sense-of-quality control unit that controls a sense of quality of theobject in the image by using the physical characteristic parameteracquired by the physical-characteristic parameter acquisition unit; and

-   -   a plurality of output units that respectively output a plurality        of pieces of information about the object whose sense of quality        has been controlled by the sense-of-quality control unit.

(2) The image processing apparatus according to (1), further including

-   -   a function information acquisition unit that acquires function        information of the plurality of output units, in which    -   the sense-of-quality control unit controls the sense of quality        of the object in the image by using the physical characteristic        parameter and the function information acquired by the function        information acquisition unit.

(3) The image processing apparatus according to (1) or (2), in which

-   -   the plurality of output units are constituted by display units        of a same type that output images about the object.

(4) The image processing apparatus according to (3), in which

-   -   the plurality of output units output the images about the object        at a same pixel position.

(5) The image processing apparatus according to (1) or (2), in which

-   -   the plurality of output units are constituted by        -   at least one display unit that outputs an image about the            object, and        -   an output unit that outputs information other than the image            about the object.

(6) The image processing apparatus according to any of (1) to (5), inwhich

-   -   the physical characteristic parameter is reflection        characteristic information indicating a reflection        characteristic of the object, and    -   the output unit outputs the reflection characteristic of the        object.

(7) The image processing apparatus according to (6), in which

-   -   one output unit outputs a specular reflection component of the        reflection characteristic of the object, and    -   another output unit outputs a diffuse reflection component of        the reflection characteristic of the object.

(8) The image processing apparatus according to (7), in which

-   -   the one output unit outputs the specular reflection component of        the reflection characteristic of the object to a front, and    -   the other output unit outputs the diffuse reflection component        of the reflection characteristic of the object to a back.

(9) The image processing apparatus according to any of (1) to (8), inwhich

-   -   the one output unit is a device having a high peak luminance,        and    -   the other output unit is a device having a low peak luminance.

(10) The image processing apparatus according to any of (1) to (9), inwhich

-   -   the physical characteristic parameter is a wavelength of the        object, and    -   the output unit performs output with respect to a wavelength of        the object.

(11) The image processing apparatus according to (10), in which

-   -   the one output unit is an electronic paper and outputs an object        color of the wavelength of the object, and    -   the other output unit is an LCD (liquid crystal display) and        outputs a light source color of the wavelength of the object.

(12) The image processing apparatus according to any of (1) to (11), inwhich

-   -   the physical characteristic parameter is a frequency component        of the object, and    -   the output unit performs output with respect to the frequency        component of the object.

(13) The image processing apparatus according to any of (1) to (12), inwhich

-   -   the physical characteristic parameter is the frequency component        of the object, and    -   the output unit performs output with respect to a texture of the        frequency component of the object.

(14) The image processing apparatus according to (13), in which

-   -   the one output unit is a tactile display and outputs the texture        of the frequency component of the object, and    -   the other output unit is an LCD (liquid crystal display) and        outputs a structure of the frequency component of the object.

(15) The image processing apparatus according to any of (1) to (14), inwhich

-   -   the physical characteristic parameter is a depth or deepness of        the object, and    -   the output unit performs output with respect to the depth or        deepness of the object.

(16) The image processing apparatus according to any of (1) to (15), inwhich

-   -   the physical characteristic parameter is information on a time        direction of the object, and    -   the output unit performs output with respect to a mobile object        and a stationary object in the time direction of the object.

(17) An image processing method, including: by an image processingapparatus,

-   -   acquiring a physical characteristic parameter regarding an        object of an image;

controlling a sense of quality of the object in the image by using theacquired physical characteristic parameter; and

-   -   respectively outputting a plurality of pieces of information        about the object whose sense of quality has been controlled, to        a plurality of display units.

REFERENCE SIGNS LIST

1 measurement and estimation block, 2 real-world modeling block, 3sense-of-quality control block, 4 rendering and retouch block, 11 imageprocessing apparatus, 21 photographing-environment informationacquisition unit, 22 photographing-environment information analyzer, 23object characteristic analyzer, 24 characteristic informationintegration unit, 25 viewing-environment information analyzer, 26characteristic control unit, 27-1 to 27-N image composition unit, 28operational environment information interpretation unit, 29prior-knowledge database, 30 additional-information generation unit,31-1 to 31-N device, 33 device information analyzer, 51 image processingapparatus

1. An image processing apparatus, comprising: a physical-characteristicparameter acquisition unit that acquires a physical characteristicparameter regarding an object of an image; a sense-of-quality controlunit that controls a sense of quality of the object in the image byusing the physical characteristic parameter acquired by thephysical-characteristic parameter acquisition unit; and a plurality ofoutput units that respectively output a plurality of pieces ofinformation about the object whose sense of quality has been controlledby the sense-of-quality control unit.
 2. The image processing apparatusaccording to claim 1, further comprising a function informationacquisition unit that acquires function information of the plurality ofoutput units, wherein the sense-of-quality control unit controls thesense of quality of the object in the image by using the physicalcharacteristic parameter and the function information acquired by thefunction information acquisition unit.
 3. The image processing apparatusaccording to claim 1, wherein the plurality of output units areconstituted by display units of a same type that output images about theobject.
 4. The image processing apparatus according to claim 3, whereinthe plurality of output units output the images about the object at asame pixel position.
 5. The image processing apparatus according toclaim 1, wherein the plurality of output units are constituted by atleast one display unit that outputs an image about the object, and anoutput unit that outputs information other than the image about theobject.
 6. The image processing apparatus according to claim 1, whereinthe physical characteristic parameter is reflection characteristicinformation indicating a reflection characteristic of the object, andthe output unit outputs the reflection characteristic of the object. 7.The image processing apparatus according to claim 6, wherein one outputunit outputs a specular reflection component of the reflectioncharacteristic of the object, and another output unit outputs a diffusereflection component of the reflection characteristic of the object. 8.The image processing apparatus according to claim 7, wherein the oneoutput unit outputs the specular reflection component of the reflectioncharacteristic of the object to a front, and the other output unitoutputs the diffuse reflection component of the reflectioncharacteristic of the object to a back.
 9. The image processingapparatus according to claim 7, wherein the one output unit is a devicehaving a high peak luminance, and the other output unit is a devicehaving a low peak luminance.
 10. The image processing apparatusaccording to claim 1, wherein the physical characteristic parameter is awavelength of the object, and the output unit performs output withrespect to a wavelength of the object.
 11. The image processingapparatus according to claim 10, wherein the one output unit is anelectronic paper and outputs an object color of the wavelength of theobject, and the other output unit is an LCD (liquid crystal display) andoutputs a light source color of the wavelength of the object.
 12. Theimage processing apparatus according to claim 1, wherein the physicalcharacteristic parameter is a frequency component of the object, and theoutput unit performs output with respect to the frequency component ofthe object.
 13. The image processing apparatus according to claim 12,wherein the physical characteristic parameter is the frequency componentof the object, and the output unit performs output with respect to atexture of the frequency component of the object.
 14. The imageprocessing apparatus according to claim 13, wherein the one output unitis a tactile display and outputs the texture of the frequency componentof the object, and the other output unit is an LCD (liquid crystaldisplay) and outputs a structure of the frequency component of theobject.
 15. The image processing apparatus according to claim 1, whereinthe physical characteristic parameter is a depth or deepness of theobject, and the output unit performs output with respect to the depth ordeepness of the object.
 16. The image processing apparatus according toclaim 1, wherein the physical characteristic parameter is information ona time direction of the object, and the output unit performs output withrespect to a mobile object and a stationary object in the time directionof the object.
 17. An image processing method, comprising: by an imageprocessing apparatus, acquiring a physical characteristic parameterregarding an object of an image; controlling a sense of quality of theobject in the image by using the acquired physical characteristicparameter; and respectively outputting a plurality of pieces ofinformation about the object whose sense of quality has been controlled,to a plurality of display units.